Selective propagation of depth measurements using stereoimaging

ABSTRACT

A depth camera assembly (DCA) determines depth information within a local area by capturing images of the local area including a local region using a plurality of imaging sensors. The local region is represented by a first set of pixels in each captured image. For each image, the DCA identifies the first set of pixels corresponding to the surface in the local region and determines a depth measurement from the DCA to the local region by comparing the first set of pixels from images captured by different imaging sensors. To determine depth measurements for second sets of pixels neighboring the first set of pixels, the DCA selectively propagates depth information from the first set of pixels to second sets of pixels satisfying one or more criteria (e.g., satisfying a threshold saturation measurement or a threshold contrast measurement).

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application Ser.No. 62/674,430, filed on May 21, 2018, which is hereby incorporated byreference in its entirety.

BACKGROUND

The present disclosure generally relates to depth estimation from imagedata, and more specifically to belief propagation of depth measurementsacross image data.

Headsets in an artificial reality system often include or are pairedwith depth estimation systems that may determine depth information ofenvironments of the artificial reality systems. Various conventionaldepth estimation systems implement semi-global matching (SGM) techniquesto propagate a depth measurement determined for a surface of theenvironment surrounding the depth estimation system to other surroundingareas in the environment surrounding the surface. Such SGM techniquesmay be computationally expensive, increasing power consumption by depthestimation systems.

SUMMARY

A depth camera assembly (DCA) determines depth information for a surfacein local area surrounding the DCA. The DCA includes at least two imagingsensors and a controller coupled to each imaging sensor. Each imagingsensor captures images of the local area surrounding the DCA. Imagescaptured by the imaging sensors include pixels corresponding todifferent portions of the local area, such as sets pixels ofcorresponding to different objects or different surfaces in the localarea surrounding the DCA.

The controller receives the captured images from the imaging sensorsand, for each image, identifies a first set of pixels corresponding tothe surface of the local area. The controller implements stereoimagingprocesses to a pair of images received from the different imagingsensors to determine a depth measurement of the surface of the localarea by comparing the first set of pixels corresponding to the surfaceidentified from each of at least two captured images received fromdifferent imaging sensors. For example, the controller applies asemi-global matching (SGM) process to an image from an imaging sensorand to a corresponding image from another imaging sensor. In the SGMprocess, when determining depth information for a target pixel, depthinformation determined for pixels neighboring the target pixel (i.e.,neighboring pixels) may be used to determine the depth information forthe target pixel. For example, the SGM process uses depth informationfor each of eight pixels adjacent to the target pixel to determine depthinformation for the target pixel.

However, propagation of depth information from neighboring pixels to atarget pixel may yield poor depth information for the target pixel whenthe target pixel has certain characteristics. This decrease in depthinformation quality when the target pixel has certain characteristicsdecreases effective usage of computing resources by the controller toleverage depth information for neighboring pixels to determine depthinformation for the target pixel. To more efficiently use computingresources, the controller may use various criteria applied to pixels incaptured images to modify a number of neighboring pixels used todetermine depth information for the target pixel. For example, thecontroller accounts for contrast or texture of pixels in the capturedimages corresponding to a region including the target pixel. In suchembodiments, if the target pixel is in a set of pixels associated with acontrast measurement greater than or lower than a threshold contrastmeasurement, the controller attenuates or halts propagation of depthinformation for neighboring pixels to the target pixel; however, if thetarget pixel is in a set of pixels having at least the thresholdcontrast measurement, the controller propagates depth information forneighboring pixels to the target pixel. For example, the controllerdetermines a contrast measurement for one or more second sets of pixelsthat are adjacent to the first set of pixels corresponding to thesurface and compares the contrast measurements for the one or moresecond sets of pixels to a threshold contrast level. For second sets ofpixels in an image with contrast measurements above the thresholdcontrast level, the controller propagates the depth measurement of thefirst set of pixels corresponding to the surface to the second sets ofpixels having contrast measurements above the threshold contrast level.However, for a second set of pixels in an image with a contrastmeasurement below the threshold contrast level, the controller halts orattenuates propagation of depth measurement for the first set of pixelsto the second set of pixels, preserving computing power and reducing lowquality depth measurements caused by low contrast. Alternatively, thecontroller measures the saturation of a second set of pixels adjacent tothe first set of pixels and halts or attenuates propagation of depthinformation from the first set of pixels to the second set of pixels inresponse to the second set of pixels having greater than a thresholdsaturation. In other embodiments, the controller halts propagation ofdepth information from the first set of pixels to pixels in the secondset of pixels that are greater than a threshold distance between thefirst set of pixels. This allows the DCA to offset certain environmentalconditions (e.g., low pixel contrast or low pixel saturation) that mayreduce the quality of depth estimation resulting from SGM techniques toa quality that does not justify the required computing power for usingthe SGM techniques, while decreasing power consumption of A DCA usingSGM techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of a headset, according to one embodiment.

FIG. 2 is a cross-section of a depth camera assembly, according to oneembodiment.

FIG. 3 is a flowchart of one embodiment of a method for propagation ofdepth information across pixels of images captured by imaging sensors ofthe depth camera assembly, according to one embodiment.

FIG. 4 is an example illustration identifying neighboring sets of pixelsin an image, according to one embodiment.

FIG. 5 is a block diagram of a system environment of a headset,according to one embodiment.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

Embodiments of the present disclosure may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereoscopic, or “stereo,” video that produces athree-dimensional effect to the viewer). Additionally, in someembodiments, artificial reality may also be associated withapplications, products, accessories, services, or some combinationthereof, that are used to, e.g., create content in an artificial realityand/or are otherwise used in (e.g., perform activities in) an artificialreality. The artificial reality system that provides the artificialreality content may be implemented on various platforms, including aheadset, a head-mounted display (HMD) connected to a host computersystem, a standalone HMD, a near-eye display (NED), a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

Headset

FIG. 1 is a perspective view of the headset 100, in accordance with oneor more embodiments. In some embodiments (as shown in FIG. 1), theheadset 100 is implemented as a NED. In alternate embodiments (not shownin FIG. 1), the headset 100 is implemented as an HMD. In general, theheadset 100 may be worn on the face of a user such that content (e.g.,media content) is presented using one or both lenses 105 of the headset100. However, the headset 100 may also be used such that media contentis presented to a user in a different manner. Examples of media contentpresented by the headset 100 include one or more images, video, audio,or some combination thereof. The headset 100 may include, among othercomponents, a frame 110, a lens 105, a depth camera assembly (DCA) 140,a passive camera assembly (PCA) 150, a position sensor 120, and an audiosystem. The audio system of the headset 100 shown in FIG. 1 includes aleft speaker 160 a, a right speaker 160 b, an array of acoustic sensors170, an audio controller 180, one or more other components, orcombination thereof. The audio system of the headset 100 is anembodiment of the audio system described above in conjunction withFIG. 1. The DCA 140 and the PCA 150 may be part of simultaneouslocalization and mapping (SLAM) sensors mounted the headset 100 forcapturing visual information of a local area surrounding some or all ofthe headset 100. While FIG. 1 illustrates the components of the headset100 in example locations on the headset 100, the components may belocated elsewhere on the headset 100, on a peripheral device paired withthe headset 100, or some combination thereof.

The headset 100 may correct or enhance the vision of a user, protect theeye of a user, or provide images to a user. The headset 100 may beeyeglasses which correct for defects in a user's eyesight. The headset100 may be sunglasses which protect a user's eye from the sun. Theheadset 100 may be safety glasses which protect a user's eye fromimpact. The headset 100 may be a night vision device or infrared gogglesto enhance a user's vision at night. The headset 100 may be a near-eyedisplay that produces artificial reality content for the user.

The frame 110 holds the other components of the headset 100. The frame110 includes a front part that holds the lens 105 and end pieces toattach to a head of the user. The front part of the frame 110 bridgesthe top of a nose of the user. The end pieces (e.g., temples) areportions of the frame 110 to which the temples of a user are attached.The length of the end piece may be adjustable (e.g., adjustable templelength) to fit different users. The end piece may also include a portionthat curls behind the ear of the user (e.g., temple tip, ear piece).

The lens 105 provides or transmits light to a user wearing the headset100. The lens 105 may be prescription lens (e.g., single vision, bifocaland trifocal, or progressive) to help correct for defects in a user'seyesight. The lens transmits ambient light to the user wearing theheadset 100. The transmitted ambient light may be altered by aprescription lens to correct for defects in the user's eyesight. Thelens 105 may be a polarized lens or a tinted lens to protect the user'seyes from the sun. The lens 105 may be one or more waveguides as part ofa waveguide display in which image light is coupled through an end oredge of the waveguide to the eye of the user. The lens 105 may includean electronic display for providing image light and may also include anoptics block for magnifying image light from the electronic display, asfurther described above in conjunction with FIG. 1.

The speakers 160 a and 160 b produce sound for user's ears. The speakers160 a and 160 b receive audio instructions from the audio controller 180to generate sounds. The left speaker 160 a may obtains a left audiochannel from the audio controller 420, and the right speaker 160 bobtains and a right audio channel from the audio controller 420. Asillustrated in FIG. 1, each speaker 160 a, 160 b is coupled to an endpiece of the frame 110 and is placed in front of an entrance to thecorresponding ear of the user. Although the speakers 160 a and 160 b areshown exterior to the frame 110, the speakers 160 a and 160 b may beenclosed in the frame 110. In some embodiments, instead of individualspeakers 160 a and 160 b for each ear, the headset 100 includes aspeaker array (not shown in FIG. 1) integrated into, e.g., end pieces ofthe frame 110 to improve directionality of presented audio content.

The DCA 140, further described below in conjunction with FIG. 2,captures depth image data describing depth information for a local areasurrounding the headset 100, such as a room. In some embodiments, theDCA 140 may include a light projector (e.g., structured light and/orflash illumination for time-of-flight), a plurality of imaging devices,and a controller. The captured data may be images captured by theimaging devices of light projected onto the local area by the lightprojector. In one embodiment, the DCA 140 includes a controller and twoor more imaging devices (e.g., cameras) that are oriented to captureimages of portions of the local area in stereo. The controller of theDCA 140 computes depth information of the local area using the capturedimages and depth determination techniques (e.g., structured light,time-of-flight, stereo imaging, etc.). Based on the depth information,the controller of the DCA 140 may determine absolute positionalinformation of the headset 100 within the local area. The controller ofthe DCA 140 may also generate a model of the local area. The DCA 140 maybe integrated with the headset 100 or may be positioned within the localarea external to the headset 100. In some embodiments, the controller ofthe DCA 140 may transmit the depth image data to a processing server viaa network or other communication channel.

The PCA 150 includes one or more passive cameras that generate color(e.g., RGB) image data. Unlike the DCA 140 that uses active lightemission and reflection, the PCA 150 captures light from the environmentof a local area to generate image data. Rather than pixel valuesdefining depth or distance from the imaging device, the pixel values ofthe image data may define the visible color of objects captured in theimaging data. In some embodiments, the PCA 150 includes a controllerthat generates the color image data based on light captured by thepassive imaging device. In some embodiments, the DCA 140 and the PCA 150share a common controller. For example, the common controller may mapeach of the one or more images captured in the visible spectrum (e.g.,image data) and in the infrared spectrum (e.g., depth image data) toeach other. In one or more embodiments, the common controller isconfigured to, additionally or alternatively, provide the one or moreimages of the local area to the audio system, or to any other suitablecomponents.

The array of acoustic sensors 170 monitor and record sound in a localarea surrounding some or all of the headset 100. As illustrated in FIG.1, the array of acoustic sensors 170 include multiple acoustic sensorswith multiple acoustic detection locations that are positioned on theheadset 100. The array of acoustic sensors 170 may provide the recordedsound as an audio stream to the audio 180.

The position sensor 120 generates one or more measurement signals inresponse to motion of the headset 100. The position sensor 120 may belocated on a portion of the frame 110 of the headset 100. The positionsensor 120 may include a position sensor, an inertial measurement unit(IMU), or both. Some embodiments of the headset 100 may or may notinclude the position sensor 120 or may include more than one positionsensor 120. In embodiments in which the position sensor 120 includes anIMU, the IMU generates IMU data based on measurement signals from theposition sensor 120. Examples of a position sensor 120 include: one ormore accelerometers, one or more gyroscopes, one or more magnetometers,another suitable type of sensor that detects motion, a type of sensorused for error correction of the IMU, or some combination thereof. Theposition sensor 120 may be located external to the IMU, internal to theIMU, or some combination thereof.

Based on the one or more measurement signals, the position sensor 120estimates a current position of the headset 100 relative to an initialposition of the headset 100. The estimated position may include alocation of the headset 100 and/or an orientation of the headset 100 orthe user's head wearing the headset 100, or some combination thereof.The orientation may correspond to a position of each ear relative to areference point. In some embodiments, the position sensor 120 uses thedepth information and/or the absolute positional information from theDCA 140 to estimate the current position of the headset 100. Theposition sensor 120 may include multiple accelerometers to measuretranslational motion (forward/back, up/down, left/right) and multiplegyroscopes to measure rotational motion (e.g., pitch, yaw, roll). Insome embodiments, an IMU rapidly samples the measurement signals andcalculates the estimated position of the headset 100 from the sampleddata. For example, the IMU integrates the measurement signals receivedfrom the accelerometers over time to estimate a velocity vector andintegrates the velocity vector over time to determine an estimatedposition of a reference point on the headset 100. The reference point isa point that may be used to describe the position of the headset 100.While the reference point may generally be defined as a point in space,in practice the reference point is defined as a point within the headset100.

The audio controller 180 provides audio instructions to the speakers 160a, 160 b for generating sound by generating audio content using a set ofacoustic parameters (e.g., a room impulse response). The audiocontroller 180 presents the audio content to appear originating from anobject (e.g., virtual object or real object) within the local area,e.g., by transforming a source audio signal using the set of acousticparameters for a current configuration of the local area.

The audio controller 180 may obtain visual information describing atleast a portion of the local area, e.g., from the DCA 140 and/or the PCA150. The visual information obtained at the audio controller 180 mayinclude depth image data captured by the DCA 140. The visual informationobtained at the audio controller 180 may further include color imagedata captured by the PCA 150. The audio controller 180 may combine thedepth image data with the color image data into the visual informationthat is communicated (e.g., via a communication module coupled to theaudio controller 180, not shown in FIG. 1) to a mapping server fordetermination of a set of acoustic parameters. In one embodiment, thecommunication module (e.g., a transceiver) may be integrated into theaudio controller 180. In another embodiment, the communication modulemay be external to the audio controller 180 and integrated into theframe 110 as a separate module coupled to the audio controller 180. Insome embodiments, the audio controller 180 generates an audio streambased on sound in the local area monitored by the array of acousticsensors 170 or by another component. The communication module coupled tothe audio controller 180 may selectively communicate the audio stream toupdate the visual model of physical spaces at the mapping server.

FIG. 2 is a cross-section 200 of the frame 110 of the headset 100 shownin FIG. 1. As shown in FIG. 2, enclosed in the frame 110 is anelectronic display 210 and an optical assembly 220 that together provideimage light to a pupil. For purposes of illustration, FIG. 2 shows across-section 200 associated with a single eye 230, but another opticalassembly 220, separate from the optical assembly 220, provides alteredimage light to another eye of the user. The frame 110 also has anoptical axis corresponding to a path along which image light propagatesthrough the frame 110.

The electronic display 210 generates image light. In some embodiments,the electronic display 210 includes an optical element that adjusts thefocus of the generated image light. The electronic display 210 displaysimage to the user in accordance with data received from a console (notshown in FIG. 2). The electronic display 210 may include a singleelectronic display or multiple electronic displays (e.g., a display foreach eye of a user). Examples of the electronic display 210 include aliquid crystal display (LCD), an organic light emitting diode (OLED)display, an inorganic light emitting diode (ILED) display, anactive-matrix organic light-emitting diode (AMOLED) display, atransparent organic light emitting diode (TOLED) display, some otherdisplay, a projector, or some combination thereof. The electronicdisplay 210 may also include an aperture, a Fresnel lens, a convex lens,a concave lens, a diffractive element, a waveguide, a filter, apolarizer, a diffuser, a fiber taper, a reflective surface, a polarizingreflective surface, or any other suitable optical element that affectsthe image light emitted from the electronic display. In someembodiments, one or more of the display block optical elements may haveone or more coatings, such as anti-reflective coatings.

The optical assembly 220 magnifies received light from the electronicdisplay 210 and corrects optical aberrations associated with the imagelight, with the corrected image light presented to a user of the headset100. At least one optical element of the optical assembly 220 may be anaperture, a Fresnel lens, a refractive lens, a reflective surface, adiffractive element, a waveguide, a filter, or any other suitableoptical element that affects the image light emitted from the electronicdisplay 210. Moreover, the optical assembly 220 may include combinationsof different optical elements. In some embodiments, one or more of theoptical elements in the optical assembly 220 may have one or morecoatings, such as anti-reflective coatings, dichroic coatings, etc.Magnification of the image light by the optical assembly 220 allowselements of the electronic display 210 to be physically smaller, weighless, and consume less power than larger displays. Additionally,magnification may increase a field-of-view of the displayed media. Forexample, the field-of-view of the displayed media is such that thedisplayed media is presented using almost all (e.g., 110 degreesdiagonal), and in some cases, all, of the user's field-of-view. In someembodiments, the optical assembly 220 is designed so its effective focallength is larger than the spacing to the electronic display 210, whichmagnifies the image light projected by the electronic display 210.Additionally, in some embodiments, the amount of magnification may beadjusted by adding or removing optical elements.

As shown in FIG. 2, the frame 110 further includes a DCA 240 fordetermining depth information of one or more objects in a local areasurrounding some or all of the headset 100. The DCA 240 includes astructured light generator 250, one or more imaging sensors 260, and acontroller 270 that may be coupled to both the structured lightgenerator 250 and each of the imaging sensors 260. The structured lightgenerator 250 emits structured light through the illumination aperture255. In various embodiments, the structured light is a pattern of dotsprojected onto a surface. The structured light generator 250 may emitthe structured light in accordance with emission instructions generatedby the controller 270.

Each of the imaging sensors 260, for example a camera, captures one ormore images of one or more objects in the local area. In someembodiments, an imaging sensor 260 captures, through the imagingaperture 265, portions of the one or more structured light patterns(e.g., polarized structured light patterns) scattered or reflected fromthe local area to illuminate objects in the local area. As furtherdescribed below in conjunction with FIG. 3, images captured by animaging sensor 260 comprise multiple pixels that may be analyzed usingstereoimaging techniques to determine depth measurements from the DCA140 to surfaces or objects in the local area that are captured in eachimage. In one embodiment, the imaging sensor 260 comprises atwo-dimensional array of pixels, with each pixel including aphotodetector, such as a photodiode.

The controller 270 determines depth information from the DCA 140 to oneor more surfaces or objects included in the local area on capturedimages of the local area including the one or more surfaces or objects.As further described below, the controller 270 implements stereoimagingtechniques to determine a depth measurement or depth information fromthe DCA 140 to different locations in the local area surrounding the DCA140 by comparing the pixels of the local region in a first image of thelocal area captured by a first imaging sensor 260 to pixels of a secondimage of the same local region of the local area captured at acorresponding time from a second imaging sensor 260. In someembodiments, the controller 270 provides the determined depthinformation to a console (not shown in FIG. 2) or an appropriate moduleof the headset 100 (e.g., a varifocal module, not shown in FIG. 2). Theconsole or the headset 100 may use the depth information to, generatecontent for presentation on the electronic display 210 or for othersuitable operations.

In various embodiments, the controller 270 may identify individualpixels of an imaging sensor 260 and retrieve digital values from eachpixel of an imaging sensor 260. To conserve computational resources, thecontroller 270 but applies one or more processes to digital valuesretrieved from specific subsets of pixels rather than to digital valuesretrieved from each pixel of the imaging sensor 260. In one embodiment,the controller 270 applies a semi-global matching (SGM) process to asubset of regions within an image rather to the image captured by theimaging sensor 260 as a whole. In such an embodiment, the controller 270identifies several regions within the local area captured in an image bygrouping pixels of the image based on a variety of measured conditions,such as the proximity of pixels to other pixels of the image, basedcontrast measurements for various pixels, or saturation measurements foreach pixel. When determining depth measurements for pixels in a capturedimage, the controller 270 propagates a depth measurement determined fora set of pixels corresponding to a first region to adjacent orsurrounding pixels corresponding to other regions that satisfy one ormore threshold conditions. Example threshold conditions include a pixelcontrast measurement, a pixel saturation measurement, or a measureddistance between pixels of different regions.

Propagating Depth Measurements Across Regions of an Area

FIG. 3 is a flowchart of one embodiment of method 300 for propagation ofdepth information across pixels of images captured by imaging sensors260 of a depth camera assembly (DCA) 140. In various embodiments, themethod 300 includes different steps than those described in conjunctionwith FIG. 3. Additionally, in various embodiments, steps of the method300 may be performed in a different order than the order described inconjunction with FIG. 3.

Each of a plurality of imaging sensors 260 of the DCA 140 captures 310an image of a local area surrounding the DCA 140. For example, theimaging sensors 260 capture images of a surface included in the localarea. In some embodiments, the image may be divided into local regions,for example tiled sections encompassing the entirety of the local areacaptured by the image. Each local region may be represented by a set ofpixels of the image. For example, a first local region corresponds to afirst set of pixels, while a second local region adjacent to the firstlocal region corresponds to a second set of pixels adjacent to the firstset of pixels. In alternate embodiments, two adjacent local regions mayinclude one or more overlapping pixels. In some embodiments, an imagingsensor 260 may capture an image under ambient lighting conditionsavailable in the environment. Alternatively, the structured lightgenerator 250 emits a structured light pattern 270 to illuminate aportion of the local area, such as a surface in the local area, toimprove the quality of the image captured by the imaging sensors 260. Animaging sensor 260 is positioned relative to other imaging sensors 260so each imaging sensor captures an image of the same area from adifferent angle. Accordingly, each imaging sensor 260 captures adifferent perspective of an area on a surface in the local area. In theembodiment illustrated in FIG. 4, a local region is represented as arectangular tile including one or more pixels, while in alternateembodiments, a local region may be represented as a tile having anysuitable geometric shape.

For each image, the controller 270 identifies 320 a first set of pixelsrepresenting the local region of the local area; for example, the firstset of pixels represents a surface of the local region in the localarea. To identify 320 the first set of pixels, the controller 270 mayoverlay an image with a pattern of tiles or grid cells to outline eachlocal region of the local area captured by the image. The controller 270analyzes each captured image to identify sets of pixels representingdifferent local regions in the image. For example, the controller 270generates a mapping that, in response to selection of a local region inan image, identifies sets of pixels representing the selected localregion in other images captured by an imaging sensor 260.

From the images captured by each of the plurality of imaging sensors,the controller 270 determines 330 a depth measurement of the localregion. The depth measurement specifies a distance between the DCA 140and a surface in the local region. In various embodiments, thecontroller 270 determines 330 the depth measurement using one or morestereoimaging processes to compare sets of pixels representing a commonlocal region in images captured by multiple imaging sensors 260. Basedon one or more differences in the sets of pixels representing the localregion in images captured by different imaging sensors 260, thecontroller 270 determines the depth measurement for the local region.

In various embodiments, a stereoimaging process, such as semi-globalmatching (SGM), determining depth information for a pixel in the firstset of pixels corresponding to the local region, the controller 270propagates 340 depth information determined 330 pixels neighboring thepixel to the pixel and accounts for the depth information for theneighboring pixels when determining 330 depth information for the pixel.Hence, the controller 270 may propagate 340 a depth measurementdetermined for a first local region to one or more adjacent or nearbyother local regions. To conserve computational resources, the controller270 may selectively propagate 340 a depth measurement for a first set ofpixels to a second set of pixels when the second set of pixels satisfyone or more criteria, but attenuate propagation of the depth measurementfor the first set of pixels (or not propagate 340) the depth measurementfor the first set of pixels to the second set of pixels when the secondset of pixels do not satisfy the one or more criteria. Example criteriafor propagating 340 the depth measurement for the first set of pixels toanother set of pixels include a threshold saturation of the other set ofpixels, a threshold contrast of the other set of pixels, and a thresholddistance between the first set of pixels and the other set of pixels.Propagation of depth measurements across an area of a captured image isfurther described with reference to FIG. 4.

In some embodiments, the controller 270 determines a contrastmeasurement for each set of pixels of an image representing a localregion (hereafter referred to as a “second set of pixels”) neighboringthe set of pixels representing the first region. A contrast is thedifference in luminance, color, distance, or other measurable quantitythat makes an object (or its representation in an image or display)distinguishable. The contrast in the images correlates to local depthperformance within a local region. For example, a contrast measurementrefers to a difference in measurement of the intensity of two groups ofpixels, one of which includes a target pixel. In one such embodiment,the controller 270 measures the contrast of a set of pixels bydetermining a ratio of a difference in luminance for the set of pixelsto an overall luminance for all pixels across the captured image. Thecontroller 270 may use a variety of contrast algorithms, such as Webercontrast, Michelson contrast, Root mean square (RMS) contrast, etc. todetermine contrast measurements for different pixels or for differentsets of pixels.

In some embodiments, the controller 270 defines a contrast at each pixel(i,j). In some embodiments, the controller 270 calculates a pixel-wisecontrast for an image using the formula:

$\begin{matrix}{{RawContrast_{i,j}} = \sqrt{\left\lbrack \frac{\left( {I_{i,{j + 1}} - I_{i,j}} \right)^{2}}{2\left( {I_{i,{j + 1}} + I_{i,j}} \right)} \right\rbrack^{2} + \left\lbrack \frac{\left( {I_{{i + 1},j} - I_{i,j}} \right)^{2}}{2\left( {I_{{i + 1},j} + I_{i,j}} \right)} \right\rbrack^{2}}} & (1)\end{matrix}$

where Ii,j is the Digital numbers of sensor pixel (i,j), and where i andj are integer values representing the i-th column and j-th row of apixel in an image sensor.

The controller 270 may subsequently calculate a sliding-window maximumof the pixel-wise contrast using:Contrast_(i,j)=max[RawContrast_(k,l)]  (2)kϵ[i−w,i+w],lϵ[j−w,j+w]where the window size is 2w+1, and (k,l) are integer values representingthe pixels within the sliding window centered at (i,j).

The controller 270 may process each pixel of a captured image todetermine a contrast measurement for each pixel of the image. Thisper-pixel contrast measurement allows the controller 270 to comparecontrast measurements for individual pixels to a threshold value, so thecontroller 270 determines on an individual pixel basis whether depthmeasurements for neighboring pixels are propagated 340 to a target pixelor group of pixels including a target pixel. Alternatively, thecontroller 270 determines a contrast measurement for a set of pixels ina local region, for example as an aggregate contrast measurement basedon individual contrast measurements for pixels of a set of pixels. Thecontroller 270 may determine average contrast values for all pixels inthe set of pixels and compare the average contrast measurement to thethreshold value. The controller 270 may also determine a count of pixelsin the set with contrast measurements equaling or exceeding thethreshold value and compare the count of pixels in the set havingcontrast measurements equaling or exceeding the threshold value to athreshold count or compare a percentage of pixels in the set havingcontrast measurements equaling or exceeding the threshold value to athreshold percentage. In the preceding example, depth information from aneighboring set of pixels is propagated 340 to a set of pixels where thecount of pixels in the set having contrast measurements equaling orexceeding the threshold value is at least the threshold count or wherethe percentage of pixels in the set having contrast measurementsequaling or exceeding the threshold value is at least the thresholdpercentage.

In different embodiments, the controller 270 may compare the contrastmeasurement for a set of pixels to a threshold value specifying an upperlimit or a lower limit. When the threshold value is a maximum contrastthreshold, a depth measurement for a first set of pixels are propagatedto second sets of pixels with contrast measurements below the maximumcontrast threshold, but are not propagated to second sets of pixels withcontrast measurements above the maximum contrast threshold. Inembodiments in which the threshold value is a minimum contrastthreshold, a depth measurement of a first set of pixels is propagated tosecond sets of pixels with contrast measurements above the minimumcontrast threshold, while the depth measurement is not be propagatedsecond sets of pixels with contrast measurements below the minimumcontrast threshold. In other embodiments, the controller 270 may useboth an upper limit and a lower limit to define a range of a contrastmeasurements for which depth measurements are propagated, so a depthmeasurement for a first set of pixels is propagated to adjacent orneighboring second sets of pixels with contrast measurements within therange of contrast measurements.

When the controller 270 compares a contrast measurement for a set ofpixels to a threshold value, the controller 270 may assign a firstweight to contrast measurements above the threshold value and a secondweight, which is less than the first weight, to contrast measurementsbelow the threshold value. When determining a depth measurement forsecond set of pixels, the controller 270 applies the weight assigned tothe contrast measurement of the second set of pixels to the depthmeasurement for the first set of pixels. For example, the controller 270multiplies the depth measurement for the first set of pixels by theweight assigned to the contrast measurement of the second set of pixels.In other embodiments, the controller 270 may categorize contrastmeasurements using more than two weights that correspond to differentvalues for contrast measurements to more particularly refine applicationof the depth measurement for the first set of pixels when determining adepth measurement for a second set of pixels. In the embodimentsdescribed above in which the controller 270 determines a contrastmeasurement for each pixel, the controller 270 may determine a depthmeasurement for each pixel by assigning a weight to each pixel based ona comparison of the pixel's contrast measurement to a threshold value.The controller 270 applies the weight assigned to a pixel to depthmeasurements from adjacent pixels to determine the depth measurement forthe pixel.

In another embodiment, the controller 270 determines a distance in theimage between a second set of pixels and a first set of pixels. In someembodiments, the controller 270 identifies a boundary enclosing eachpixel of a set and determines a distance, for example a perpendiculardistance, between the boundaries of different sets of pixels. Thecontroller 270 propagates a depth measurement for the first set ofpixels to second sets of pixels that are within a threshold distance ofthe first set of pixels. In alternate embodiments, the controller 270may identify a distance, for example a perpendicular distance, betweeneach pixel of the first set of pixels and each pixel of a second set ofpixels. The controller 270 may identify particular pixels of the secondset of pixels within the threshold distance of at least one pixel of thefirst set or within the threshold distance of a threshold number ofpixels of the first set and propagate the depth measurement of the firstset of pixels to the identified particular pixels of the second set ofpixels. Alternatively, the controller 270 propagates the depthmeasurement for the first set of pixels to an second set of pixels inresponse to at least a threshold amount (e.g., number, percentage) orpixels of the second set of pixels being within the threshold distanceof at least a threshold amount (e.g., number, percentage) of pixels ofthe first set

In some embodiments, the controller 270 determines a saturationmeasurement for each pixel of the image. As described herein, asaturation measurement characterizes intensity of one or more colors inpixels of the imaging sensors 260. For example, a pixel having anintensity of one or more colors above a threshold value specified duringconfiguration of the DCA 140 is characterized as saturated. For example,if an imaging sensor includes ret-blue-green (RGB) pixels, a pixelhaving above a threshold color value is identified as a saturated pixel.The controller 270 may compare a saturation measurement for a pixel or asaturation measurement determined for a set of pixels to a thresholdsaturation value and propagate 340 a depth measurement determined for anadjacent pixel or for an adjacent set of pixels to the pixel or to theset of pixels in response to the saturation measurement for a pixel or asaturation measurement determined for a set of pixels being less than orequal to the threshold saturation measurement. As further describedabove regarding contrast measurements, the controller 270 may determinea saturation measurement for individual pixels or for sets of pixels.The methods described above for determining an aggregate contrastmeasurement may also be used to determine an aggregate saturationmeasurement for a set of pixels.

As described above regarding propagation of depth measurements based oncontrast measurements, the controller 270 may compare a saturationmeasurement for an individual pixel or for a set of pixels to athreshold maximum value, to a threshold minimum value, or to a thresholdrange of values. Hence, depth measurements for a pixel are propagated toa pixel or to multiple pixels adjacent to the pixel satisfying thoseconditions. As further described above, the controller 270 may assign aweight to a pixel or to a set including multiple pixels based on thesaturation measurement for the pixel or the set of pixels. For example,a first weight is assigned to a set of pixels if the saturationmeasurement for the set of pixels is greater than a threshold, while adifferent second weight is assigned to the set of pixels if thesaturation measurement for the set of pixels is less than the threshold.The weight assigned to the pixel is applied to depth measurements forpixels, or for additional sets of pixels, adjacent to the set of pixelsthat are propagated to the set of pixels to determine a depthmeasurement for the set of pixels.

FIG. 4 is an example illustration of local regions of an image includingpixels neighboring a first set of pixels, according to one embodiment.In the illustrated embodiment of FIG. 4, an image captured by an imagesensor, for example imaging sensor 260, is divided by a controller, forexample controller 270, into several sets of pixels—sets 405, 410, 415,420, 425, 430, 435, 440, 445, 450, 455, 460, 465, 470, 475, and 480 inFIG. 4. In the example shown by FIG. 4, sets of pixels compriserectangular regions of the captured area including multiple pixels.However, in other embodiments, the sets of pixels may have any suitableshape or size. In FIG. 4, a set 405 of pixels represents a first set ofpixels for which a depth measurement has been determined through one ormore stereoimaging processes. Also in FIG. 4, sets 410-480 of pixelsrepresent second sets of pixels neighboring the first set 405. In theexample of FIG. 4, sets 410, 415, 420, 430, 435, 445, and 450 of pixelsare each associated with a contrast measurement within a first range ofcontrast measurements and sets 425, 440, 455, 460, 465, 470, 475, and480 are each associated with a contrast measurement within a differentsecond range of contrast measurements. For illustrative purposes, setsof pixels associated with contrast measurements within the first rangeare represented with a perpendicular line cross-hatchings and setsassociated with contrast measurements within the second range arerepresented with diagonal line cross-hatchings. In other embodiments,sets 410-480 of pixels may be categorized based on threshold distancemeasurements, based on threshold saturation measurements, or any othersuitable information (which may be instead of or in combination with thecontrast measurements described above).

In one embodiment, a controller, for example the controller 270,executes instructions to propagate the depth measurement determined forset 405 of pixels to neighboring sets of pixels having contrastmeasurements within the first threshold range. In such an embodiment,the depth measurement for set 405 of pixels is propagated to sets 410,415, 420, 430, 435, 445, and 450, but is not propagated to sets 425,440, 455, 460, 465, 470, 475, and 480 of pixels. In one or morealternative embodiments, the controller propagates the depth measurementdetermined for set 405 of pixels to neighboring sets of pixels havingcontrast measurements satisfying the second threshold range, so thecontroller propagates the depth measurement for set 405 of pixels tosets 425, 440, 455, 460, 465, 470, 475, and 480. In the precedingexample, the controller does not propagate the depth measurement for set405 of pixels to sets 410, 415, 420, 430, 435, 445, and 450 of pixels.In some embodiments, when propagating the depth measurement for set 405of pixels to sets 410-480 of pixels, the controller applies a firstweight to the depth measurement for set 405 of pixels when propagatingthe depth measurement to sets 410, 415, 420, 430, 435, 445, and 450 ofpixels and applies a different second weight to the depth measurementfor set 405 of pixels 405 when propagating the depth measurement to sets425, 440, 455, 460, 465, 470, 475, and 480.

In other embodiments, sets of pixels may be categorized based onthreshold distance measurements or based on threshold saturationmeasurements instead of or in combination with the contrast measurementsdescribed above. Implementations involving saturation measurements maybe consistent with the description above. In implementations involvingthe distance measurements between sets of pixels, the sets 410, 415,420, 430, 435, 445, 450, and 455 of pixels may be associated withdistances relative to set 405 of pixels satisfying a threshold distancemeasurement, while distances of sets 425, 440, 460, 465, 470, 475, and480 of pixels relative to set 405 of pixels do not satisfy the thresholddistance measurement. Thus, the depth measurement determined for set 405would be propagated to sets 410, 415, 420, 430, 435, 445, 450, and 455of pixels, but not propagated to sets 425, 440, 460, 465, 470, 475, and480 of pixels.

System Environment

FIG. 5 is a block diagram of one embodiment of a headset system 500 inwhich a console 510 operates. The headset system 500 may operate in a VRsystem environment, an AR system environment, a MR system environment,or some combination thereof. The headset system 500 shown by FIG. 5comprises a headset 505 and an input/output (I/O) interface 515 that iscoupled to the console 510. While FIG. 5 shows an example headset 505including one headset 505 and on I/O interface 515, in other embodimentsany number of these components may be included in the headset 505. Forexample, there may be multiple headsets 505 each having an associatedI/O interface 515, with each headset 505 and I/O interface 515communicating with the console 510. In alternative configurations,different and/or additional components may be included in the headset505. Additionally, functionality described in conjunction with one ormore of the components shown in FIG. 5 may be distributed among thecomponents in a different manner than described in conjunction with FIG.5 in some embodiments. For example, some or all of the functionality ofthe console 510 is provided by the headset 505.

The headset 505 is a head-mounted display that presents content to auser comprising virtual and/or augmented views of a physical, real-worldenvironment with computer-generated elements (e.g., two-dimensional (2D)or three-dimensional (3D) images, 2D or 3D video, sound, etc.). In someembodiments, the presented content includes audio that is presented viaan external device (e.g., speakers and/or headphones) that receivesaudio information from the headset 505, the console 510, or both, andpresents audio data based on the audio information. The headset 505 maycomprise one or more rigid bodies, which may be rigidly or non-rigidlycoupled together. A rigid coupling between rigid bodies causes thecoupled rigid bodies to act as a single rigid entity. In contrast, anon-rigid coupling between rigid bodies allows the rigid bodies to moverelative to each other. An embodiment of the headset 505 is the headset100 described above in conjunction with FIG. 1.

The headset 505 includes a depth camera assembly 520, an electronicdisplay 525, an optical assembly 530, one or more position sensors 535,an IMU 540, an eye tracking system 545, and a varifocal module 550. Someembodiments of the headset 505 have different components than thosedescribed in conjunction with FIG. 5. Additionally, the functionalityprovided by various components described in conjunction with FIG. 5 maybe differently distributed among the components of the headset 505 inother embodiments.

The depth camera assembly 520 includes multiple imaging sensors thatcapture data describing a local area surrounding some or all of theheadset 505 and depth information associated with the local area. Thedepth camera assembly 520 computes depth information for one or morelocal regions within the local area using the data captured by theimaging sensors. When determining depth information for different localregions, one or more stereoimaging processes used by the depth cameraassembly 520 propagate depth information determined for a local regionto neighboring or adjacent local regions to determine depth informationfor the neighboring or adjacent local regions. To more efficiently usecomputing resources and power when determining depth information, thedepth camera assembly 520 accounts for characteristics—such as contrastmeasurements or saturation measurements—of local regions to selectivelypropagate depth information determined for a local region to other localregions having specific characteristics, while attenuating propagationof depth information determined for the local region to other localregions that do not have the specific characteristics, as furtherdescribed above in conjunction with FIGS. 3 and 4. In some embodiments,the DCA also comprises a structured light generator to furtherilluminate the surface of the area captured by the imaging sensors.

In some embodiments, the DCA 520 implements stereoimaging techniques todetermine a depth measurement for a local region to compare the imagesof a common area as captured by different imaging sensors at variouspositions. The stereoimaging techniques may propagate depth informationfrom the adjacent local regions when determining depth information forthe adjacent local regions. As further described above, the DCA 520selectively propagates depth information from the local region toadjacent local regions satisfying one or more criteria. For example, theDCA 520 limits propagation of depth information from the local region toadjacent local regions having a contrast measurement satisfying one ormore criteria, to adjacent local regions having a saturation measurementsatisfying one or more criteria, or to adjacent local regions having adistance from the local region satisfying one or more criteria. In thepreceding example, the DCA 520 attenuates depth information for thelocal region that is propagated to adjacent local regions that do notsatisfy one or more criteria, as further described above in conjunctionwith FIGS. 3 and 4.

The electronic display 525 displays 2D or 3D images to the user inaccordance with data received from the console 510. In variousembodiments, the electronic display 525 comprises a single electronicdisplay or multiple electronic displays (e.g., a display for each eye ofa user). Examples of the electronic display 525 include: a liquidcrystal display (LCD), an organic light emitting diode (OLED) display,an inorganic light emitting diode (ILED) display, an active-matrixorganic light emitting diode (AMOLED) display, a transparent organiclight emitting diode (TOLED) display, some other display, or somecombination thereof.

The optical assembly 530 magnifies image light received from theelectronic display 525, corrects optical errors associated with theimage light, and presents the corrected image light to a user of theheadset 505. The optical assembly 530 includes a plurality of opticalelements. Example optical elements included in the optical assembly 530include: an aperture, a Fresnel lens, a convex lens, a concave lens, afilter, a reflecting surface, or any other suitable optical element thataffects image light. Moreover, the optical assembly 530 may includecombinations of different optical elements. In some embodiments, one ormore of the optical elements in the optical assembly 530 may have one ormore coatings, such as partially reflective or anti-reflective coatings.

Magnification and focusing of the image light by the optical assembly530 allows the electronic display 525 to be physically smaller, weighless and consume less power than larger displays. Additionally,magnification may increase the field of view of the content presented bythe electronic display 525. For example, the field of view of thedisplayed content is such that the displayed content is presented usingalmost all (e.g., approximately 110 degrees diagonal), and in some casesall, of the user's field of view. Additionally in some embodiments, theamount of magnification may be adjusted by adding or removing opticalelements.

In some embodiments, the optical assembly 530 may be designed to correctone or more types of optical error. Examples of optical error includebarrel or pincushion distortions, longitudinal chromatic aberrations, ortransverse chromatic aberrations. Other types of optical errors mayfurther include spherical aberrations, chromatic aberrations or errorsdue to the lens field curvature, astigmatisms, or any other type ofoptical error. In some embodiments, content provided to the electronicdisplay 525 for display is pre-distorted, and the optical assembly 530corrects the distortion when it receives image light from the electronicdisplay 525 generated based on the content.

The IMU 540 is an electronic device that generates data indicating aposition of the headset 505 based on measurement signals received fromone or more of the position sensors 535 and from depth informationreceived from the depth camera assembly 520. A position sensor 535generates one or more measurement signals in response to motion of theheadset 505. Examples of position sensors 535 include: one or moreaccelerometers, one or more gyroscopes, one or more magnetometers,another suitable type of sensor that detects motion, a type of sensorused for error correction of the IMU 540, or some combination thereof.The position sensors 535 may be located external to the IMU 540,internal to the IMU 540, or some combination thereof.

Based on the one or more measurement signals from one or more positionsensors 535, the IMU 540 generates data indicating an estimated currentposition of the headset 505 relative to an initial position of theheadset 505. For example, the position sensors 535 include multipleaccelerometers to measure translational motion (forward/back, up/down,left/right) and multiple gyroscopes to measure rotational motion (e.g.,pitch, yaw, roll). In some embodiments, the IMU 540 rapidly samples themeasurement signals and calculates the estimated current position of theheadset 505 from the sampled data. For example, the IMU 540 integratesthe measurement signals received from the accelerometers over time toestimate a velocity vector and integrates the velocity vector over timeto determine an estimated current position of a reference point on theheadset 505. Alternatively, the IMU 540 provides the sampled measurementsignals to the console 510, which interprets the data to reduce error.The reference point is a point that may be used to describe the positionof the headset 505. The reference point may generally be defined as apoint in space or a position related to the orientation and position ofthe headset 505.

The IMU 540 receives one or more parameters from the console 510. Theone or more parameters are used to maintain tracking of the headset 505.Based on a received parameter, the IMU 540 may adjust one or more IMUparameters (e.g., sample rate). In some embodiments, certain parameterscause the IMU 540 to update an initial position of the reference pointso it corresponds to a next position of the reference point. Updatingthe initial position of the reference point as the next calibratedposition of the reference point helps reduce accumulated errorassociated with the current position estimated the IMU 540. Theaccumulated error, also referred to as drift error, causes the estimatedposition of the reference point to “drift” away from the actual positionof the reference point over time. In some embodiments of the headset505, the IMU 540 may be a dedicated hardware component. In otherembodiments, the IMU 540 may be a software component implemented in oneor more processors.

In some embodiments, the eye tracking system 545 is integrated into theheadset 505. The eye tracking system 545 determines eye trackinginformation associated with an eye of a user wearing the headset 505.The eye tracking information determined by the eye tracking system 545may comprise information about an orientation of the user's eye, i.e.,information about an angle of an eye-gaze. In some embodiments, the eyetracking system 545 is integrated into the optical assembly 530. Anembodiment of the eye tracking system 545 may comprise an illuminationsource such as a dot array illuminator as described herein and animaging device (camera).

n some embodiments, the varifocal module 550 is further integrated intothe headset 505. The varifocal module 550 may be coupled to the eyetracking system 545 to obtain eye tracking information determined by theeye tracking system 545. The varifocal module 550 may be configured toadjust focus of one or more images displayed on the electronic display525, based on the determined eye tracking information obtained from theeye tracking system 545. In this way, the varifocal module 550 canmitigate vergence-accommodation conflict in relation to image light. Thevarifocal module 550 can be interfaced (e.g., either mechanically orelectrically) with at least one of the electronic display 525 and atleast one optical element of the optical assembly 530. Then, thevarifocal module 550 may be configured to adjust focus of the one ormore images displayed on the electronic display 525 by adjustingposition of at least one of the electronic display 525 and the at leastone optical element of the optical assembly 530, based on the determinedeye tracking information obtained from the eye tracking system 545. Byadjusting the position, the varifocal module 550 varies focus of imagelight output from the electronic display 525 towards the user's eye. Thevarifocal module 550 may be also configured to adjust resolution of theimages displayed on the electronic display 525 by performing foveatedrendering of the displayed images, based at least in part on thedetermined eye tracking information obtained from the eye trackingsystem 545. In this case, the varifocal module 550 provides appropriateimage signals to the electronic display 525. The varifocal module 550provides image signals with a maximum pixel density for the electronicdisplay 525 only in a foveal region of the user's eye-gaze, whileproviding image signals with lower pixel densities in other regions ofthe electronic display 525. In one embodiment, the varifocal module 550may utilize the depth information obtained by the depth camera assembly520 to, e.g., generate content for presentation on the electronicdisplay 525.

The I/O interface 515 is a device that allows a user to send actionrequests and receive responses from the console 510. An action requestis a request to perform a particular action. For example, an actionrequest may be an instruction to start or end capture of image or videodata or an instruction to perform a particular action within anapplication. The I/O interface 515 may include one or more inputdevices. Example input devices include: a keyboard, a mouse, a gamecontroller, or any other suitable device for receiving action requestsand communicating the action requests to the console 510. An actionrequest received by the I/O interface 515 is communicated to the console510, which performs an action corresponding to the action request. Insome embodiments, the I/O interface 515 includes an IMU 540 thatcaptures calibration data indicating an estimated position of the I/Ointerface 515 relative to an initial position of the I/O interface 515.In some embodiments, the I/O interface 515 may provide haptic feedbackto the user in accordance with instructions received from the console510. For example, haptic feedback is provided when an action request isreceived, or the console 510 communicates instructions to the I/Ointerface 515 causing the I/O interface 515 to generate haptic feedbackwhen the console 510 performs an action.

The console 510 provides content to the headset 505 for processing inaccordance with information received from one or more of: the depthcamera assembly 520, the headset 505, and the I/O interface 515. In theexample shown in FIG. 5, the console 510 includes an application store555, a tracking module 560, and an engine 565. Some embodiments of theconsole 510 have different modules or components than those described inconjunction with FIG. 5. Similarly, the functions further describedbelow may be distributed among components of the console 510 in adifferent manner than described in conjunction with FIG. 5.

The application store 555 stores one or more applications for executionby the console 510. An application is a group of instructions, that whenexecuted by a processor, generates content for presentation to the user.Content generated by an application may be in response to inputsreceived from the user via movement of the headset 505 or the I/Ointerface 515. Examples of applications include: gaming applications,conferencing applications, video playback applications, or othersuitable applications.

The tracking module 560 calibrates the headset system 500 using one ormore calibration parameters and may adjust one or more calibrationparameters to reduce error in determination of the position of theheadset 505 or of the I/O interface 515. For example, the trackingmodule 560 communicates a calibration parameter to the depth cameraassembly 520 to adjust the focus of the depth camera assembly 520 tomore accurately determine positions of structured light elementscaptured by the depth camera assembly 520. Calibration performed by thetracking module 560 also accounts for information received from the IMU540 in the headset 505 and/or an IMU 540 included in the I/O interface515. Additionally, if tracking of the headset 505 is lost (e.g., thedepth camera assembly 520 loses line of sight of at least a thresholdnumber of structured light elements), the tracking module 560 mayre-calibrate some or all of the headset system 500.

The tracking module 560 tracks movements of the headset 505 or of theI/O interface 515 using information from the depth camera assembly 520,the one or more position sensors 535, the IMU 540 or some combinationthereof. For example, the tracking module 550 determines a position of areference point of the headset 505 in a mapping of a local area based oninformation from the headset 505. The tracking module 560 may alsodetermine positions of the reference point of the headset 505 or areference point of the I/O interface 515 using data indicating aposition of the headset 505 from the IMU 540 or using data indicating aposition of the I/O interface 515 from an IMU 540 included in the I/Ointerface 515, respectively. Additionally, in some embodiments, thetracking module 560 may use portions of data indicating a position orthe headset 505 from the IMU 540 as well as representations of the localarea from the depth camera assembly 520 to predict a future location ofthe headset 505. The tracking module 560 provides the estimated orpredicted future position of the headset 505 or the I/O interface 515 tothe engine 565.

The engine 565 generates a 3D mapping of the area surrounding some orall of the headset 505 (i.e., the “local area”) based on informationreceived from the headset 505. In some embodiments, the engine 565determines depth information for the 3D mapping of the local area basedon information received from the depth camera assembly 520 that isrelevant for techniques used in computing depth. The engine 565 maycalculate depth information using one or more techniques in computingdepth from one or more polarized structured light patterns. In variousembodiments, the engine 565 uses the depth information to, e.g., updatea model of the local area, and generate content based in part on theupdated model.

The engine 565 also executes applications within the headset system 500and receives position information, acceleration information, velocityinformation, predicted future positions, or some combination thereof, ofthe headset 505 from the tracking module 560. Based on the receivedinformation, the engine 565 determines content to provide to the headset505 for presentation to the user. For example, if the receivedinformation indicates that the user has looked to the left, the engine565 generates content for the headset 505 that mirrors the user'smovement in a virtual environment or in an environment augmenting thelocal area with additional content. Additionally, the engine 565performs an action within an application executing on the console 510 inresponse to an action request received from the I/O interface 515 andprovides feedback to the user that the action was performed. Theprovided feedback may be visual or audible feedback via the headset 505or haptic feedback via the I/O interface 515.

In some embodiments, based on the eye tracking information (e.g.,orientation of the user's eye) received from the eye tracking system545, the engine 565 determines resolution of the content provided to theheadset 505 for presentation to the user on the electronic display 525.The engine 565 provides the content to the headset 505 having a maximumpixel resolution on the electronic display 525 in a foveal region of theuser's gaze, whereas the engine 565 provides a lower pixel resolution inother regions of the electronic display 525, thus achieving less powerconsumption at the headset 505 and saving computing cycles of theconsole 510 without compromising a visual experience of the user. Insome embodiments, the engine 565 can further use the eye trackinginformation to adjust where objects are displayed on the electronicdisplay 525 to prevent vergence-accommodation conflict.

Additional Configuration Information

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

Some portions of this description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are commonly used bythose skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, areunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, or the like. Furthermore, it has alsoproven convenient at times, to refer to these arrangements of operationsas modules, without loss of generality. The described operations andtheir associated modules may be embodied in software, firmware,hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, and/or it may comprise a general-purpose computingdevice selectively activated or reconfigured by a computer programstored in the computer. Such a computer program may be stored in anon-transitory, tangible computer readable storage medium, or any typeof media suitable for storing electronic instructions, which may becoupled to a computer system bus. Furthermore, any computing systemsreferred to in the specification may include a single processor or maybe architectures employing multiple processor designs for increasedcomputing capability.

Embodiments may also relate to a product that is produced by a computingprocess described herein. Such a product may comprise informationresulting from a computing process, where the information is stored on anon-transitory, tangible computer readable storage medium and mayinclude any embodiment of a computer program product or other datacombination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the patent rights. It istherefore intended that the scope of the patent rights be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: capturing, by each of aplurality of imaging sensors, an image of a local area including a localregion, the local region represented by a first set of pixels in eachcaptured image; identifying, for each image, the first set of pixelscorresponding to the local region; determining, from the image capturedby each of the plurality of imaging sensors, a depth measurement of thelocal region, the depth measurement determined by a comparison betweenthe first set of pixels from the image captured by each of the pluralityof imaging sensors; determining, for each of one or more second sets ofpixels neighboring the first set of pixels, a contrast measurementassociated with each second set of pixels; and responsive to determininga contrast measurement for a second set of pixels is above a thresholdcontrast measurement, propagating the depth measurement of the localregion corresponding to the first set of pixels to a local regioncorresponding to the second set of pixels having the contrastmeasurement above the threshold contrast.
 2. The method of claim 1,wherein the depth measurement is determined by performing semi-globalmatching (SGM) techniques to the first set of pixels from the imagecaptured by each of the plurality of imaging sensors, the SGM techniquespropagating the depth measurement of the local region corresponding tothe first set of pixels to one or more sets of pixels adjacent to thefirst set of pixels.
 3. The method of claim 1, further comprising:responsive to determining the contrast measurement for the second set ofpixels is below the threshold contrast measurement, attenuatingpropagation of the depth measurement of the local region correspondingto the first set of pixels to the local region corresponding to thesecond set of pixels.
 4. The method of claim 3, wherein attenuatingpropagation of the depth measurement of the local region to the secondset of pixels comprises: withholding propagation of the depthmeasurement of the local region corresponding to the first set of pixelsto the local region corresponding to the second set of pixels.
 5. Themethod of claim 1, further comprising: assigning a first weight to eachsecond set of pixels having a contrast measurement above the thresholdcontrast measurement; determining a depth measurement for each secondset of pixels having the contrast measurement above the thresholdcontrast measurement by applying the first weight to the depthmeasurement of the local region corresponding to the first set ofpixels; assigning a second weight to each second set of pixels having acontrast measurement below the threshold contrast measurement; anddetermining a depth measurement for each second set of pixels having acontrast measurement below the threshold contrast measurement byapplying the second weight to the depth measurement of the local regioncorresponding to the first set of pixels.
 6. The method of claim 1,further comprising: measuring a distance in the images captured by theimaging sensors between each pixel of the first set of pixels and eachpixel of the second set of pixels; and propagating the depth measurementof the local region corresponding to the first set of pixels to one ormore pixels of the second set with a measured distance within athreshold distance of at least one pixel of the first set.
 7. The methodof claim 1, further comprising: determining, for each of one or moresecond sets of pixels neighboring the first set of pixels, a saturationmeasurement associated with each second set of pixels; and responsive todetermining the saturation measurement of the second set of pixels isabove a threshold saturation measurement, attenuating propagation of thedepth measurement of the local region corresponding to the first set ofpixels to the local region corresponding to the second set of pixels. 8.The method of claim 7, wherein attenuating propagation of the depthmeasurement of the local region to the second set of pixels comprises:withholding propagation of the depth measurement of the local regioncorresponding to the first set of pixels to the local regioncorresponding to the second set of pixels.
 9. A depth camera assembly(DCA) comprising: a plurality of imaging sensors configured to capturean image of a local area including a local region, the local regionrepresented by a first set of pixels in each captured image; and acontroller configured to: identify, for each image, the first set ofpixels corresponding to the local region; determine, from the imagecaptured by each of the plurality of imaging sensors, a depthmeasurement of the local region, the depth measurement determined by acomparison between the first set of pixels from the image captured byeach of the plurality of imaging sensors; determine, for each of one ormore second sets of pixels neighboring the first set of pixels, acontrast measurement associated with each second set of pixels; andresponsive to determining a contrast measurement for a second set ofpixels is above a threshold contrast measurement, propagate the depthmeasurement of the local region corresponding to the first set of pixelsto a local region corresponding to the second set of pixels having thecontrast measurement above the threshold contrast.
 10. The DCA of claim9, wherein the controller is further configured to determine the depthmeasurement by performing semi-global matching (SGM) techniques to thefirst set of pixels from the image captured by each of the plurality ofimaging sensors, the SGM techniques propagating the depth measurement ofthe local region corresponding to the first set of pixels to one or moresets of pixels adjacent to the first set of pixels.
 11. The DCA of claim9, wherein the controller is further configured to: responsive todetermining the contrast measurement for the second set of pixels isbelow the threshold contrast measurement, attenuate propagation of thedepth measurement of the local region corresponding to the first set ofpixels to the local region corresponding to the second set of pixels.12. The DCA of claim 9, wherein the controller is further configured to:assign a first weight to each second set of pixels having a contrastmeasurement above the threshold contrast measurement; determine a depthmeasurement for each second set of pixels having the contrastmeasurement above the threshold contrast measurement by applying thefirst weight to the depth measurement of the local region correspondingto the first set of pixels; assign a second weight to each second set ofpixels having a contrast measurement below the threshold contrastmeasurement; and determine a depth measurement for each second set ofpixels having a contrast measurement below the threshold contrastmeasurement by applying the second weight to the depth measurement ofthe local region corresponding to the first set of pixels.
 13. The DCAof claim 9, wherein the controller is further configured to: measure adistance in the images captured by the imaging sensors between eachpixel of the first set of pixels and each pixel of the second set ofpixels; and propagate the depth measurement of the local regioncorresponding to the first set of pixels to one or more pixels of thesecond set with a measured distance within a threshold distance of atleast one pixel of the first set.
 14. The DCA of claim 9, wherein thecontroller is further configured to: determine, for each of one or moresecond sets of pixels neighboring the first set of pixels, a saturationmeasurement associated with each second set of pixels; and responsive todetermining the saturation measurement of the second set of pixels isabove a threshold saturation measurement, attenuate propagation of thedepth measurement of the local region corresponding to the first set ofpixels to the local region corresponding to the second set of pixels.15. A non-transitory computer readable medium comprising computerprogram instructions that when executed by a computer processor causethe processor to: capture, by each of a plurality of imaging sensors, animage of a local area including a local region, the local regionrepresented by a first set of pixels in each captured image; identify,for each image, the first set of pixels corresponding to the localregion; determine, from the image captured by each of the plurality ofimaging sensors, a depth measurement of the local region, the depthmeasurement determined by a comparison between the first set of pixelsfrom the image captured by each of the plurality of imaging sensors;determine, for each of one or more second sets of pixels neighboring thefirst set of pixels, a contrast measurement associated with each secondset of pixels; and responsive to determining a contrast measurement fora second set of pixels is above a threshold contrast measurement,propagate the depth measurement of the local region corresponding to thefirst set of pixels to a local region corresponding to the second set ofpixels having the contrast measurement above the threshold contrast. 16.The non-transitory computer readable storage medium of claim 15, whereinthe computer program instructions further cause the processor todetermine a depth measurement by performing semi-global matching (SGM)techniques to the first set of pixels from the image captured by each ofthe plurality of imaging sensors, the SGM techniques propagating thedepth measurement of the local region corresponding to the first set ofpixels to one or more sets of pixels adjacent to the first set ofpixels.
 17. The non-transitory computer readable storage medium of claim15, wherein the computer program instructions further cause theprocessor to: responsive to determining the contrast measurement for thesecond set of pixels is below the threshold contrast measurement,attenuate propagation of the depth measurement of the local regioncorresponding to the first set of pixels to the local regioncorresponding to the second set of pixels.
 18. The non-transitorycomputer readable storage medium of claim 15, wherein the computerprogram instructions further cause the processor to: assign a firstweight to each second set of pixels having a contrast measurement abovethe threshold contrast measurement; determine a depth measurement foreach second set of pixels having the contrast measurement above thethreshold contrast measurement by applying the first weight to the depthmeasurement of the local region corresponding to the first set ofpixels; assign a second weight to each second set of pixels having acontrast measurement below the threshold contrast measurement; anddetermine a depth measurement for each second set of pixels having acontrast measurement below the threshold contrast measurement byapplying the second weight to the depth measurement of the local regioncorresponding to the first set of pixels.
 19. The non-transitorycomputer readable storage medium of claim 15, wherein the computerprogram instructions further cause the processor to: measure a distancein the images captured by the imaging sensors between each pixel of thefirst set of pixels and each pixel of the second set of pixels; andpropagate the depth measurement of the local region corresponding to thefirst set of pixels to one or more pixels of the second set with ameasured distance within a threshold distance of at least one pixel ofthe first set.
 20. The non-transitory computer readable storage mediumof claim 15, wherein the computer program instructions further cause theprocessor to: determine, for each of one or more second sets of pixelsneighboring the first set of pixels, a saturation measurement associatedwith each second set of pixels; and responsive to determining thesaturation measurement of the second set of pixels is above a thresholdsaturation measurement, attenuate propagation of the depth measurementof the local region corresponding to the first set of pixels to thelocal region corresponding to the second set of pixels.